{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Rent Listing Inqueries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第二步：调整树的参数：max_depth & min_child_weight\n",
    "(粗调，参数的步长为2；下一步是在粗调最佳参数周围，将步长降为1，进行精细调整)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先 import 必要的模块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold,train_test_split\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "train = pd.read_csv(dpath +\"RentListingInquries_FE_train.csv\")\n",
    "test = pd.read_csv(dpath +\"RentListingInquries_FE_test.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每类样本分布不是很均匀，所以交叉验证时也考虑各类样本按比例抽取"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Y = train['interest_level']\n",
    "X = train.drop([\"interest_level\"], axis=1)\n",
    "X = np.array(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.8, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train=X\n",
    "y_train=Y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各类样本不均衡，交叉验证是采用StratifiedKFold，在每折采样时各类样本按比例采样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第一轮参数调整得到的n_estimators最优值（231），其余参数继续默认值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用交叉验证评价模型性能时，用scoring参数定义评价指标。评价指标是越高越好，因此用一些损失函数当评价指标时，需要再加负号，如neg_log_loss，neg_mean_squared_error 详见sklearn文档：http://scikit-learn.org/stable/modules/model_evaluation.html#log-loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': [4, 5, 6, 7, 8], 'min_child_weight': [4, 5, 6, 7, 8]}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "max_depth = [4,5,6,7,8]\n",
    "min_child_weight = [4,5,6,7,8]\n",
    "param_test = dict(max_depth=max_depth, min_child_weight=min_child_weight)\n",
    "param_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.59398, std: 0.00354, params: {'max_depth': 4, 'min_child_weight': 4},\n",
       "  mean: -0.59406, std: 0.00338, params: {'max_depth': 4, 'min_child_weight': 5},\n",
       "  mean: -0.59420, std: 0.00340, params: {'max_depth': 4, 'min_child_weight': 6},\n",
       "  mean: -0.59417, std: 0.00317, params: {'max_depth': 4, 'min_child_weight': 7},\n",
       "  mean: -0.59448, std: 0.00395, params: {'max_depth': 4, 'min_child_weight': 8},\n",
       "  mean: -0.59081, std: 0.00315, params: {'max_depth': 5, 'min_child_weight': 4},\n",
       "  mean: -0.59061, std: 0.00344, params: {'max_depth': 5, 'min_child_weight': 5},\n",
       "  mean: -0.59066, std: 0.00343, params: {'max_depth': 5, 'min_child_weight': 6},\n",
       "  mean: -0.59056, std: 0.00387, params: {'max_depth': 5, 'min_child_weight': 7},\n",
       "  mean: -0.59104, std: 0.00423, params: {'max_depth': 5, 'min_child_weight': 8},\n",
       "  mean: -0.58993, std: 0.00333, params: {'max_depth': 6, 'min_child_weight': 4},\n",
       "  mean: -0.58971, std: 0.00381, params: {'max_depth': 6, 'min_child_weight': 5},\n",
       "  mean: -0.58924, std: 0.00308, params: {'max_depth': 6, 'min_child_weight': 6},\n",
       "  mean: -0.58918, std: 0.00355, params: {'max_depth': 6, 'min_child_weight': 7},\n",
       "  mean: -0.58964, std: 0.00332, params: {'max_depth': 6, 'min_child_weight': 8},\n",
       "  mean: -0.58931, std: 0.00359, params: {'max_depth': 7, 'min_child_weight': 4},\n",
       "  mean: -0.59004, std: 0.00381, params: {'max_depth': 7, 'min_child_weight': 5},\n",
       "  mean: -0.59017, std: 0.00387, params: {'max_depth': 7, 'min_child_weight': 6},\n",
       "  mean: -0.59045, std: 0.00359, params: {'max_depth': 7, 'min_child_weight': 7},\n",
       "  mean: -0.59072, std: 0.00436, params: {'max_depth': 7, 'min_child_weight': 8},\n",
       "  mean: -0.59397, std: 0.00357, params: {'max_depth': 8, 'min_child_weight': 4},\n",
       "  mean: -0.59444, std: 0.00292, params: {'max_depth': 8, 'min_child_weight': 5},\n",
       "  mean: -0.59321, std: 0.00425, params: {'max_depth': 8, 'min_child_weight': 6},\n",
       "  mean: -0.59312, std: 0.00412, params: {'max_depth': 8, 'min_child_weight': 7},\n",
       "  mean: -0.59290, std: 0.00349, params: {'max_depth': 8, 'min_child_weight': 8}],\n",
       " {'max_depth': 6, 'min_child_weight': 7},\n",
       " -0.5891755428605883)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=231,  #第一轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch = GridSearchCV(xgb, param_grid = param_test, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch.fit(X_train , y_train)\n",
    "\n",
    "gsearch.grid_scores_, gsearch.best_params_,     gsearch.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([449.04684653, 451.98111539, 453.70241814, 437.43559976,\n",
       "        435.24887252, 525.73881211, 519.96915975, 513.21573329,\n",
       "        481.75560055, 488.48986325, 574.5626369 , 584.53310719,\n",
       "        660.71569943, 649.01598697, 628.70998425, 622.50680614,\n",
       "        638.51578026, 655.76787124, 639.38404016, 622.67328634,\n",
       "        720.07816906, 711.91940131, 702.91546588, 690.77545543,\n",
       "        608.39833803]),\n",
       " 'mean_score_time': array([1.20279727, 1.19598103, 1.15065999, 1.00056095, 1.04748554,\n",
       "        1.37184806, 1.4667007 , 1.28882728, 1.36452894, 1.4047358 ,\n",
       "        1.93715134, 1.95891004, 2.4659616 , 2.08043189, 1.54711418,\n",
       "        1.84901705, 2.26743035, 2.19654088, 1.89343491, 1.83528056,\n",
       "        2.49122472, 2.16004448, 2.13347368, 2.04012532, 1.65293074]),\n",
       " 'mean_test_score': array([-0.59398108, -0.5940582 , -0.59419883, -0.59417488, -0.5944835 ,\n",
       "        -0.59081094, -0.59060523, -0.59066137, -0.59055764, -0.59103781,\n",
       "        -0.58993172, -0.58970689, -0.58924005, -0.58917554, -0.58963937,\n",
       "        -0.58931071, -0.59004395, -0.59016778, -0.59044746, -0.59071763,\n",
       "        -0.59396663, -0.59443874, -0.59320934, -0.59311965, -0.5929001 ]),\n",
       " 'mean_train_score': array([-0.55144085, -0.55208534, -0.55269398, -0.55313596, -0.5538815 ,\n",
       "        -0.51817482, -0.52000624, -0.52137734, -0.52285522, -0.5238576 ,\n",
       "        -0.47982387, -0.48322418, -0.4855575 , -0.48828025, -0.49060272,\n",
       "        -0.43571138, -0.44127108, -0.44577415, -0.45091593, -0.45439197,\n",
       "        -0.38943173, -0.39850161, -0.40692105, -0.41331691, -0.41884861]),\n",
       " 'param_max_depth': masked_array(data=[4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 7, 7, 7,\n",
       "                    7, 7, 8, 8, 8, 8, 8],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_min_child_weight': masked_array(data=[4, 5, 6, 7, 8, 4, 5, 6, 7, 8, 4, 5, 6, 7, 8, 4, 5, 6,\n",
       "                    7, 8, 4, 5, 6, 7, 8],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False, False, False, False, False, False, False, False,\n",
       "                    False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'max_depth': 4, 'min_child_weight': 4},\n",
       "  {'max_depth': 4, 'min_child_weight': 5},\n",
       "  {'max_depth': 4, 'min_child_weight': 6},\n",
       "  {'max_depth': 4, 'min_child_weight': 7},\n",
       "  {'max_depth': 4, 'min_child_weight': 8},\n",
       "  {'max_depth': 5, 'min_child_weight': 4},\n",
       "  {'max_depth': 5, 'min_child_weight': 5},\n",
       "  {'max_depth': 5, 'min_child_weight': 6},\n",
       "  {'max_depth': 5, 'min_child_weight': 7},\n",
       "  {'max_depth': 5, 'min_child_weight': 8},\n",
       "  {'max_depth': 6, 'min_child_weight': 4},\n",
       "  {'max_depth': 6, 'min_child_weight': 5},\n",
       "  {'max_depth': 6, 'min_child_weight': 6},\n",
       "  {'max_depth': 6, 'min_child_weight': 7},\n",
       "  {'max_depth': 6, 'min_child_weight': 8},\n",
       "  {'max_depth': 7, 'min_child_weight': 4},\n",
       "  {'max_depth': 7, 'min_child_weight': 5},\n",
       "  {'max_depth': 7, 'min_child_weight': 6},\n",
       "  {'max_depth': 7, 'min_child_weight': 7},\n",
       "  {'max_depth': 7, 'min_child_weight': 8},\n",
       "  {'max_depth': 8, 'min_child_weight': 4},\n",
       "  {'max_depth': 8, 'min_child_weight': 5},\n",
       "  {'max_depth': 8, 'min_child_weight': 6},\n",
       "  {'max_depth': 8, 'min_child_weight': 7},\n",
       "  {'max_depth': 8, 'min_child_weight': 8}],\n",
       " 'rank_test_score': array([20, 21, 23, 22, 25, 14, 11, 12, 10, 15,  6,  5,  2,  1,  4,  3,  7,\n",
       "         8,  9, 13, 19, 24, 18, 17, 16]),\n",
       " 'split0_test_score': array([-0.58908273, -0.5899621 , -0.58944453, -0.59009732, -0.58927452,\n",
       "        -0.58555307, -0.58469222, -0.58487938, -0.58377671, -0.58417608,\n",
       "        -0.58533814, -0.58402606, -0.58458037, -0.58395419, -0.58405566,\n",
       "        -0.5829434 , -0.58373509, -0.58464934, -0.58500795, -0.58549789,\n",
       "        -0.58824983, -0.59003666, -0.58865969, -0.58742927, -0.5874375 ]),\n",
       " 'split0_train_score': array([-0.55287179, -0.55380275, -0.55445922, -0.5542171 , -0.55552639,\n",
       "        -0.52051221, -0.52127609, -0.52330739, -0.52393819, -0.52499523,\n",
       "        -0.48046989, -0.4847596 , -0.48694039, -0.48932325, -0.49232224,\n",
       "        -0.4375367 , -0.44356401, -0.44964866, -0.45380682, -0.45563514,\n",
       "        -0.39014346, -0.39776695, -0.40593118, -0.41320326, -0.41778082]),\n",
       " 'split1_test_score': array([-0.59204848, -0.59205423, -0.59219931, -0.59289549, -0.59179067,\n",
       "        -0.58954054, -0.5900463 , -0.58953927, -0.58936453, -0.58840385,\n",
       "        -0.58915531, -0.588682  , -0.58780839, -0.58745944, -0.58965289,\n",
       "        -0.58868216, -0.58869294, -0.58954024, -0.58849728, -0.58789601,\n",
       "        -0.59251335, -0.59208437, -0.59027014, -0.58941231, -0.59091592]),\n",
       " 'split1_train_score': array([-0.55140477, -0.55217468, -0.55283259, -0.55367228, -0.55415509,\n",
       "        -0.51711169, -0.51979513, -0.52116049, -0.52332514, -0.52284106,\n",
       "        -0.47948632, -0.48306294, -0.48506094, -0.48735406, -0.49008108,\n",
       "        -0.43469541, -0.44067748, -0.44429603, -0.45026986, -0.45344484,\n",
       "        -0.38975562, -0.3983534 , -0.40809343, -0.41294201, -0.42043369]),\n",
       " 'split2_test_score': array([-0.59269537, -0.59218504, -0.5931925 , -0.59209969, -0.59325306,\n",
       "        -0.59111152, -0.59019866, -0.5906455 , -0.5912057 , -0.59237935,\n",
       "        -0.58874008, -0.58833778, -0.58942865, -0.58830162, -0.58912126,\n",
       "        -0.59002011, -0.59101802, -0.58945268, -0.59107699, -0.58959543,\n",
       "        -0.59364775, -0.59540478, -0.59150639, -0.59505198, -0.59438181]),\n",
       " 'split2_train_score': array([-0.55116829, -0.55211933, -0.55273206, -0.55321892, -0.55356796,\n",
       "        -0.51876219, -0.52109665, -0.52208339, -0.52366669, -0.52450967,\n",
       "        -0.47936697, -0.48373126, -0.4869861 , -0.49023224, -0.49175323,\n",
       "        -0.4365418 , -0.44089128, -0.44496737, -0.45095033, -0.45553926,\n",
       "        -0.39143793, -0.40048182, -0.40894468, -0.4146817 , -0.41988177]),\n",
       " 'split3_test_score': array([-0.59764554, -0.59879442, -0.59794892, -0.59820939, -0.5993739 ,\n",
       "        -0.5932383 , -0.59415304, -0.59440207, -0.59399943, -0.59456889,\n",
       "        -0.59088376, -0.59217868, -0.59042043, -0.59225597, -0.59107243,\n",
       "        -0.59120942, -0.59144824, -0.59042694, -0.59183762, -0.59240871,\n",
       "        -0.59710891, -0.59752042, -0.59500765, -0.59499069, -0.59399653]),\n",
       " 'split3_train_score': array([-0.55094075, -0.55136516, -0.55203453, -0.55235912, -0.55346037,\n",
       "        -0.51792094, -0.51901001, -0.52124418, -0.52224469, -0.52366399,\n",
       "        -0.47930865, -0.4812778 , -0.48432654, -0.48671411, -0.48968352,\n",
       "        -0.43474326, -0.44019248, -0.44425706, -0.44992746, -0.45365884,\n",
       "        -0.38880913, -0.39856494, -0.40599307, -0.41259238, -0.41805666]),\n",
       " 'split4_test_score': array([-0.59843465, -0.59729622, -0.59821011, -0.59757354, -0.59872665,\n",
       "        -0.59461243, -0.59393696, -0.5938416 , -0.59444301, -0.59566227,\n",
       "        -0.59554302, -0.59531166, -0.59396386, -0.59390793, -0.59429603,\n",
       "        -0.59369979, -0.59532707, -0.5967717 , -0.59581907, -0.59819241,\n",
       "        -0.59831465, -0.59714828, -0.60060508, -0.59871567, -0.59777024]),\n",
       " 'split4_train_score': array([-0.55081867, -0.55096479, -0.55141153, -0.55221236, -0.5526977 ,\n",
       "        -0.51656708, -0.5188533 , -0.51909124, -0.52110142, -0.52327805,\n",
       "        -0.48048754, -0.48328932, -0.48447354, -0.48777757, -0.48917352,\n",
       "        -0.43503972, -0.44103019, -0.44570162, -0.44962516, -0.45368176,\n",
       "        -0.38701251, -0.39734094, -0.40564288, -0.41316523, -0.41809013]),\n",
       " 'std_fit_time': array([ 3.28290956,  2.45976566,  2.28373269,  7.44085937,  7.29068103,\n",
       "         1.53307992,  5.87227593,  1.41734823,  0.80937711,  8.47800964,\n",
       "        17.26372931, 55.32452004, 63.4980455 , 64.17016022, 67.35452848,\n",
       "        10.06049233,  5.29668197, 10.28000758,  4.72490128,  5.80856378,\n",
       "         8.46079019,  6.4328615 ,  5.25288166,  2.04034461, 64.4800626 ]),\n",
       " 'std_score_time': array([0.05779056, 0.05771839, 0.09166333, 0.08208785, 0.08299485,\n",
       "        0.12493285, 0.21844523, 0.03976595, 0.05524631, 0.09630929,\n",
       "        0.22376529, 0.23836234, 0.30759013, 0.35770005, 0.0313805 ,\n",
       "        0.0303767 , 0.44148009, 0.2941233 , 0.04287514, 0.03630085,\n",
       "        0.12823855, 0.08070175, 0.06976044, 0.05104595, 0.09229983]),\n",
       " 'std_test_score': array([0.00353955, 0.00338288, 0.00339901, 0.00317482, 0.00394508,\n",
       "        0.00315251, 0.00343886, 0.00342923, 0.00386695, 0.00423495,\n",
       "        0.00333199, 0.00381352, 0.00308226, 0.0035455 , 0.00332209,\n",
       "        0.00358631, 0.00380671, 0.00387372, 0.0035936 , 0.0043632 ,\n",
       "        0.00356855, 0.0029221 , 0.00424636, 0.00411511, 0.00348968]),\n",
       " 'std_train_score': array([0.00074302, 0.00097293, 0.0010214 , 0.00076419, 0.0009442 ,\n",
       "        0.00138477, 0.00101665, 0.00137946, 0.00104888, 0.00079071,\n",
       "        0.00053776, 0.00113431, 0.00117389, 0.00130088, 0.00121984,\n",
       "        0.00113507, 0.00118115, 0.00200771, 0.00151122, 0.00097987,\n",
       "        0.00147489, 0.00108019, 0.00133748, 0.00071611, 0.00108835])}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.589176 using {'max_depth': 6, 'min_child_weight': 7}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZoAAAELCAYAAADgPECFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzsvXl0XOWVr/3sKpVKs2RrsGxL8oQx\nNh5ko2DMYGbCYENIHILT9IV0uHSv9L0k5AsNCdyGdELHobmBkF6dDp00JOE2hCYDtoGAMRgngCE2\nyBhswCO2PEiyrXmqUtX+/jilUkkqjVapStJ+1qp1pvecs89R6fxq73efd4uqYhiGYRixwhVvAwzD\nMIyxjQmNYRiGEVNMaAzDMIyYYkJjGIZhxBQTGsMwDCOmmNAYhmEYMcWExjAMw4gpJjSGYRhGTDGh\nMQzDMGJKUjxOKiITgd8A04EDwA2qWhOlXQDYEVo8qKrXhtZfCvwLjlA2Areo6h4RmQb8J5APnARu\nUtWK/uzJy8vT6dOnn+JVGYZhjC+2bdt2XFXz+2sn8RiCRkQeBE6q6hoRuRuYoKp3RWnXqKoZUdZ/\nAlynqrtE5GvA2ap6i4j8N7BeVX8pIpcAX1HVv+7PnrKyMt26deupX5hhGMY4QkS2qWpZf+3iFTq7\nDvhlaP6XwOcGub8CWaH5bOBIaH4esDE0/1roPIZhGEYciZfQTFLVowChaUEv7VJEZKuIbBGRSDG6\nFXhBRCqAvwbWhNZvB74Qmr8eyBSR3OE33zAMwxgoMRMaEXlFRD6I8hmMl1EScsu+DDwiIrNC6+8A\nrlbVIuBx4Eeh9d8CLhSR94ALgcNAey/23RYSsa3V1dVDuUTDMAxjAMQsGUBVL+ttm4hUishkVT0q\nIpOBql6OcSQ03Scim4DFIlIPLFLVt0PNfgP8MaL950PnyAC+oKp1vRz7MeAxcPpohnCJhmEYxgCI\nV+hsLXBzaP5m4LnuDURkgoh4Q/N5wHnATqAGyBaR00NNLwd2dbQTkY5r+jZOBpphGIYRR+KS3ozT\np/KMiHwVOAh8EUBEyoC/U9VbgbnAz0QkiCOIa1R1Z6jd/wR+G9pWA/xN6LgXAT8QEQU2A38/cpdk\nGIZhRCMu6c2JhqU3G4ZhDJ6BpjfHy6MxDMPoE1XFF/TRFmjDF3Cmbe1tzjRiXXhboA1/0M+M7Bmc\nmXsmaZ60eF+CEcKExjCMqKgq7cH2Hg/2qA/5oDPf2t4aVQB6a9+XcPiCviHb7hY3cybOoTS/lMUF\niyktKKUwvXAY744xGExoDCOBCQQDA36w9/YrP/LBPSAhCHYuBzV4SvYnuZLwur143V6S3cnh+Y7l\njOQMJron9mjTvW1v27qvc4ubT2o+obyqnO3V2/n9nt/zXx/9FwCT0iZRWlBKaX4ppQWlzJk4B4/L\nMxx/JqMfTGgMI4Gobq6mvLqc96reY3vVdnae3El7MOqrYANCEFKSUqI+5L1uL94kL1nerK4PbVdy\nj336e8inuHu2T3Yl43a5h/HuDIzC9EKWFy0HoD3YHhae8qpyyqvLeenASwCkuFM4M+/MsNezKH8R\nOSk5I27veMCSAbBkACM+BIIB9tTuobyqnPeq36O8qpzDjYcBSHYlMz9vPgvzFzIhZULPh73LGxaK\nvgQgSZIQkThfaWJxrOkY26u3h72eXSd20a6OmE/Pmt7F65mRPQOX2CD3vTHQZAATGoYuNO0nTtDw\nykayrvws7uzsGFhmjCWa/E1sr97O9qrtvFf1Hu8ff58mfxMAuSm54b6E0oJS5k2ch8dtYZ2RoKW9\nhQ+Pf0h5dTnbq7ZTXl1ObVstAJnJmSzKX+T8bfJLmZ8335IMIjChGQRDFZra3/6Wo/fci3g8ZFx0\nIVkrVpJx0YW4vN4YWGmMJlSVI01HeK/qvXDYZnftboIaRBBmT5gdDteUFpRSlFFknkeCoKp8Wv8p\n5dXlYa9nT+0ewEkyOH3C6V28nsnpk8ft386EZhAMVWhUldYPd1K/bh11zz9P4PhxXJmZZF35WbJW\nriStrAxxmds9HvAH/Hx08iNHWEIPqOoWZwy9tKQ0FuYvDP8qXpi/kIzkHtUvjASmrq2OHcd3hPvO\n3j/+Pi3tLQAUpBawqKDT6zlj4hnjxhs1oRkEw9FHo+3tNG15m/p1a6nf8Ara3EzS5Mlkr1hB9rUr\n8c6ePUzWGolAbWttWFDKq8v54PgHtAXaAJiaMbUz3FJQyuyc2XHpFDdiR3uwnd01u7t4PR39a163\nlzNzzwx7PYsKFjExZWKcLY4NJjSDYLiTAYLNzTRsfJW6dWtpeuNNCATwzp1L9ooVZK24Bs+kScN2\nLiP2qCr76/eHQ2DvVb3HgfoDACRJEnNz53YRloK03qpeGGOZquaq8A+P7hmD07KmhUNtpfmlzMyZ\nOSaSDExoBkEss87aT5yg/oUXqVu3jtb33wcR0s5ZSvbKa8m84nLcGRZCSTQiO4c7Hhx1bc4g4Nne\n7C4PjDPzziQ1KTXOFhuJSGt7KztP7Ozi9ZxsPQlApieThQULw9+lBXkLSPekx9niwWNCMwhGKr25\nbf9+6tetp27dOvyHDiFeL5mXXkLWypVknH8+4hkfcd1Eo+OX6HtV70VNd43MBpueNX1M/BI1Rh5V\n5VDDofB7UuVV5eyt3YuiuMTFnAlzwskhpQWlTEmfkvBJBiY0g2Ck36NRVVrKy6lft476F14kUFuL\nOyeHrKuvImvlSlJLSxP+CzZaCQQD7K7d3UVYusfWO4RlUf4iJqRMiLPFxlim3lfPjuodYa/n/er3\naW5vBiA/NT/8PSwtKGXuxLkku5PjbHFXTGgGQTxf2FSfj8Y33qB+3ToaNr6KtrXhKS4me+VKslau\nwDtjRlzsGis0+hp5v/r98K/IaP/IHeGLuRPnjptsISMx6XiJNzJ7sftLvIsKFjlJBvmLyE2Nb6V6\nE5pBkCgjAwQaG2l4eQN169bSvOVtUCVl4UJHdK6+iqTc+H6pEh1V5XDj4bCn8l7Ve+yu2R0OTczO\nmR0OSywuWDwqQhOGUd1cHR7JoLy6nJ0nduIP+gEoySzp4vWclnPaiIZ2TWgGQaIITST+ykrq1z9P\n3fr1tO3aBW436eed6yQRXHoJrjR7O9kf8LPr5K4uwnK85TgA6Z50FuY5764sKljEwjx7d8UYG7QF\n2th1Ylc4/FteXR5OMsjwZLAof1HY61mYvzCmSQYmNIMgEYUmktZPPnGSCNavp/3oUSQtjazLLyNr\n5bWkn7MUSRofY6PWtNaEBaW8qpwPT3zY5d2V0oJSFucvDv+ys3dXjPGAqlLRUNElSzKaJ9+Rgj81\nY+qwefImNIMg0YWmAw0Gad661Uki+ONLBBsacOfnkX311WStvJaUM+eNmVBQUIMcqDvQJVYdfnfF\nlcS8ifO6vI2dn5YfX4MNI4Fo8DWw4/iO8LtfkePq5aXmhfslF+UvYl7uvCEnGZjQDILRIjSRBNva\naHz9dSeJYNPr4PeTPHMm2deuJGvFCpKLiuJt4qBoaW/hg+MfdL7wVr09/O5Kjjcn/Ib14oLFnJl7\nJilJKXG22DBGDx1JBpF9PYcaDgGw+ozVfGfpd4Z0XBOaQTAahSaSQG0t9S+9TN26tbRs3QZA6lln\nOUkEV34Wd07i1diobKrsdPWryvno5Efhd1dmZM8Ieyod766MFU/NMBKF4y3H2V69nSnpU5ibO3dI\nxzChGQSjXWgi8VUcpn6981Kob+9e8HjIWL6c7JUrybj4ohEbWVpVqWmrobKpksrmSo41HaOyuZKK\nhgrer36fI01HAKf41Py8+Z3jQlnxKcMYNZjQDIKxJDQdqCptu3ZRt3Yddc+vJ1DtjCyd+dkryF55\nLWmfGfrI0kENcrL1JJVNlRxrPhYWk7CgNFVS1VzVo+a7W9xMSpvEmXnOS5GLCxYzZ8Ice3fFMEYp\nJjSDYCwKTSQaCNC0ZQv1a9fRsGEDweZmkgoLyV65gqyVK0k5/fRw20AwwInWE1HFo2O5srmyR3nh\nJFcSk9ImOZ/0SRSmFTIpfVJ4XWF6IRNTJlommGGMIUxoBsFYF5oO2oPtVJ88xPENL+J/cSMpW3ch\nQeVEUSblpdm8PjfAHk9NuK+kg2RXcqdopHcKR+TyxJSJNgaYYYwzBio04+MFjHGAP+inurna8Ti6\n9Yt0hLiOtxwnqEFnh0sh6xwXyz92c+FOH5eur+CS5+HkvKk0X1KG99ILKcifTmF6ITneHOuMNwxj\nyJjQjAJ8AR9VzVW9ikhlcyXHW46jdPVOU5NSw57HssnLnPlu4ays5CxEBN+BA9StW0/yunX4f/Ic\n8tgfybjkYpJWXgvnnwfJiTWYn2EYowcLnRHf0FlboI2qpiqONR/rIR4dgtIxvEQkGZ6MHuGr7v0j\nGZ6MQXsiqkrr9u3UrVtP/QsvEKipwZ2dTebVV5G98lpSF9vI0oZhOFgfzSCIldC0tLd07USP4o3U\ntNX02C8rOauH5xEpIgVpBSMybpf6/c7I0mvX0bBxY8TI0ivIWrES70wbWdowxjMJLTQiMhH4DTAd\nOADcoKo9nrgiEgB2hBYPquq1ofWXAA8BycA24Kuq2i7OT+0fA1cDzcAtqvpuf/YMVWhqWmv46ORH\nUUNZlc2V4TfbI8nx5nQKR6Q3EiEiaZ7EGzAz0NhIw4ZXqF+3lqa3tjgjSy9Y0DmydF5evE00DGOE\nSXSheRA4qaprRORuYIKq3hWlXaOqZnRb5wI+BS5V1U9E5J+AT1X1FyJyNfC/cYRmKfBjVV3anz1D\nFZo/7v8jd26+M7w8MWVi1MysjmlBWsGYGDrFX1lF/QsvULduLW07QyNLn3su2deuJPPSS21kacMY\nJyS60HwMXKSqR0VkMrBJVedEaRdNaPKBt1T1tNDyBcC3VfVqEflZ6FhPdT9PX/YMVWhOtJxgf91+\nCtMdTyTRqt+NBG27d1O3bj1169fRfsQZWTrzskvJXnkt6cvOGTcjSxvGeCTR05sndTz8Q2JT0Eu7\nFBHZCrQDa1T1D8BxwCMiZaq6FVgFFIfaTwUORexfEVrXp9AMldzU3LhXuIs33tmzKfjmHeR/4+u0\nbNvmJBH88Y/Ur12HOy+PrFASQcr8My2JwDDGKTETGhF5BSiMsumeQRymRFWPiMhM4FUR2aGqe0Xk\nRuBhEfECL+MIEUC0J1lUl01EbgNuAygpKRmESUY0xOUi7TOfIe0zn2HSvfc4I0uvXUftU09T86tf\nkzxjhjOy9FVX4Zk6FfHYsDOGMV5I6NBZt32eANar6rPd1l8B3KqqN4x06Mzon0BdHfUvvUT92nU0\nR9xjSU3FnZGBKyurc5qZgSszC3dWJq6MzK7TzEzcmZ1TSUszD8kw4kyih87WAjcDa0LT57o3EJEJ\nQLOqtolIHnAe8GBoW4GqVoU8mruAByKO+79E5GmcZIC6/kTGiC3u7Gwm3HADE264Af/hwzRu3kx7\nTQ3BhkYCDfUEGxoJNtQTqK3Ff/AggcZGgvX1qN/fz4HdjkBlZuLKysSd0cu0Q5yysrqKVkaGeVWG\nMULES2jWAM+IyFeBg8AXAUSkDPg7Vb0VmAv8TESCgAunj2ZnaP87RWRFaP1PVfXV0PoXcDLO9uCk\nN39lpC7I6B/P1KlMWL16QG2DbW0E6+sJdAhRfQPBxoau04YGAg2dU//BQ7SGloONjf2eQ1JTu3hJ\nXabmVRnGsGEvbGKhs7GIBgIEm5oihKmeYGOjM43wprp4VQ2ONzVoryorC1dmBu7MfqbmVRljjEQP\nnRlGTBG3G3dWFu6srCEfo4dXFZ6GvKhYeFXdw35ZWXgmTyG5pBhPcTHuHBvg1Bh9mNAYRi+4vF5c\n+fkk5ecPaf8heVU1tfg/7b2vypWZSXJxMZ6SktC0mOTiEpJLikkqLBxyMTvDiCUmNIYRI4bFq2pu\nxn/4ML5Dh/AdPIj/4CF8hw7RtmsXDRs3QoQQiceDp7i4pxCVlOApKsJlI3AbccKExjASGFdaGt7Z\ns/HOnt1jmwYC+I8ew3/oIL5PD+I71ClEzX/5C8Hm5s7GIiQVFvbwgjyh6amIoWH0hwmNYYxSxO0m\nuWgqyUVTSV+2rMs2VSVw8qTjBR06hO/gIUeQDh6icdPrBI4f79LenZ0dNRznKSkhKT/fQnLGKWFC\nYxhjEBEhKTeXpNxcWLy4x/ZgUxO+ioqIcJwzbdmxg/qXXoJAoPNYXi+e4qIu4pPcIUpTpiAWkjP6\nwYTGMMYhrvR0UubMIWVOzwE51O/Hf+QIvggB8h06hP/gQZq2bEFbWiIO5MIzeXLUcJynuAR3RvoI\nXpWRqJjQGIbRBfF4SJ42jeRp03psU1Xaq6t7hON8hw7SsGEDgZquZaXcEydGTU5ILi7GnZdnqdrj\nBBMawzAGjIjgKSjAU1BA2lln9dgeaGgIi5DjDTlC1LxtK/Xr10PEC+KSlkZyUVF0b2jKFCsxMYaw\nv6RhGMOGOzMT97x5pMyb12Nb0OfDX3G4ixfkP3gI3/4DNG3+E+rzRRzIjWfKFMcLmlbSVYiKi6y4\n3ijDhMYwjBHBlZyMd+YMvDNn9NimwSDtVVVRs+RaXniRYF3Xsuju/DxHfLqF4zwlJbgnTLCQXIJh\nQmMYRtwRlwtPYSGewkI4++we2wN1dRHi4whQR3JC+3NdB393pafjKSpyMuWKikMZc8V4iorxTJ2C\ny+sdqcsyQpjQGIaR8Lizs0ldkE3qgvk9tgVbW/FXVHQKUcVhxys6cICmP/0ZbWvrbCxCUkFBdBEq\nmuq8M2Te0LBjQmMYxqjGlZKC97TT8J52Wo9t4Sy5igpHfCoq8B+qwFdxiKa33qL9D5Vd2ktKCp6i\nqSERKia5uCgsQslF1jc0VExoDMMYs0RmybFkSY/twbY2/IcPOx7RoUNhEfIfqqD5nXe6DuMDuPPy\nnEy57iJUXEzSpEk2gkIvmNAYhhF/2n3ga4S2htC0EXwNoWlvy93bN4K/GXJnQ8k5ULLMmaZN7PW0\nLq8X78yZeGfO7LFNVQnU1EQVoZZt26h//nkIBsPtxePBM3VqVBHyFBfjzsiIya0bDVjhM6zw2Yig\nCs0nofkEuNzgToYkb8TUC/ZrcPQQDHR74HcXgoZ+tndbDvj6PyeAywPeDEjODE0zIqaZznep8kM4\n/C4EQyNb582Bacs6hSdnGgxDP4z6fPiPHQuLkL/iEL5DnSG6YH19l/bunJxOEZpa1Nk/VFyMp7Bw\nVL43NNDCZyY0mNCcEqrQUgMNx6DxmDNtOBqaRnwaj/X/MHElOYKTlNxt6o0iTKFpUkp00RrQMfpp\n50oalgdSQqDq/NrvVQgGuexv7v+cAOJyRKCLIEQTigEuJw0wY8zfAkfeg4NvwcEtcPBtaAulSGdO\nDnk85zrTSWc6P36GmUBdXbhPqIsIHa7Af/gItLd3Ng6/N9QhQp1eUXJxEa7s7IRMUjChGQQmNFFQ\nhda6XgTkKDRUdi4H2nru782GzELInOT8Y2cWOtO0PNCgs097myM+UadtTjily7S/9qEPw/Wdlr4F\naUjiNoT2XQRiICGlKCEmX6Nz3weCJ72fh37Ie+ixnN5zmyctMcQ6GITqXfDpmyHheQvqDzvbkjOh\n+OxOj2fqWZAc205/bW+nvbLSEZ/uInSogsDJk13auzIzHQ8oigjFc2BTE5pBMK6ERtX5ddohGI2V\n3QQkwgtpb+m5f3JmSDQiPxFCkjHJmU+O02CKqhBsH6JonWK73rYF2/u3+1Rwe/sQgEF6D8npMfl1\nn5DUHuoUnYNboGonoE54bkppZz9P8TmQnjuipgUam/AfDolPNxHyV1R0HUVBhKTJhdFFqLgY98SJ\nMfOGTGgGwZgRmrbGroLRmyfib+q5rye9m2j0IiLe8duhOWSCfXlwAxA3T1rvQpKc4XhExqnTUgOH\n3ukUnsPbOsO9ead3DbdNmB43T02Dwc6BTUMi5D9cEZ5vr67u0l5SU3tmynX0D02diislZci2mNAM\ngoQXGl9ThKfRXUQi1vkae+6blApZkyGju3B0W/Zmjvx1GUYi42/t2s9zaIsTTgbn/6nD45m2DCbN\nTxhPMNjSEi7/7a843CNJoUuZByD3ttso+OYdQzrXQIVm9KU5jCX8LT3Fokc46xi01ffcNymlUygm\nzYfTLo8Qjoh+EW9WQsTI/X4/FRUVtLa2xtsUYxyQkpJCUVERHo9n6AfxpDgiMi1UvTQYhOqPIhIM\n3oKdf3C2JWdC8Wci+nnKYt7P0xuu1NQ+X2ANnDjRRYRSFiyMuU3m0RADj8bfGvI4ovR/RIazWut6\n7utO7upp9PBEQkKSkpMQAjJQ9u/fT2ZmJrm5uQmZPWOMHVSVEydO0NDQwIwZPQfwHFZqD8GhtzvF\np/JDnH6eJJhc2vV9nvS82NoSB8yjGQkqP4Qdz/YUkJaanm1dns5wVd5smLE81HE+uauQpE4YVQIy\nUFpbW5k+fbqJjBFzRITc3Fyqu/VVxIScYuezYJWz3FLbtZ/nnf+At/7V2Rb5Ium0ZTBhxpj8X4+G\nCc2pcHI/vPloyOuYBBNnwrRzIzrPI7yR1Anj/oVEExljpIjbdy01B06/wvmAk9BxpLxTeHatg/d+\n7WzLmNTV45m0ANxj85E8Nq9qpDj9Sri3etwLiGEYvZDkhZKlzgecfp7jH3fr5wmVOUjOgKKIfp6i\nsvi9JjDM2BPyVHAnmciMImpra/m3f/u3Ie37yCOP0Nw8wLfh+2Dr1q3cfvvtp3ycDm655RaeffbZ\nHuuPHDnCqlVOOGfTpk2sWLEi6v7Tp0/n+PHjw2ZPB7feeis7d+7ss01vth84cID/+q//6nPfQCDA\n4sWLe72uhMXlgoK5UPY38PnH4Bs74I6d8IVfwKLV0HQcNv0AfnUt/KAYHrsYXrrH8YQaRyAUGCPM\nozHGDR1C87WvfW3Q+z7yyCPcdNNNpJ3iMPFlZWWUlfXbd3rKTJkyJepDfKT4+c9/PuR9O4Tmy1/+\ncq9tfvzjHzN37lzq66NkZI42sqc6fTwd/TytdXDoL51ez19+HtHPc1pEuG2ZE64fBSHpuAiNiEwE\nfgNMBw4AN6hqjx50EQkAO0KLB1X12tD6S4CHgGRgG/BVVW0XkTOAx4ElwD2q+lCML8UYAt9d9yE7\njwzvA2LelCzuW3lmn23uvvtu9u7dS2lpKZdffjkFBQU888wztLW1cf311/Pd736XpqYmbrjhBioq\nKggEAvyf//N/qKys5MiRI1x88cXk5eXx2muvRT1+RkYGf//3f88rr7zChAkT+Od//mf+4R/+gYMH\nD/LII49w7bXXsmnTJh566CHWr1/P/fffz8GDB9m3bx8HDx7kG9/4Rp/ezq9+9SseeughRISFCxfy\n6187sf7Nmzfzox/9iGPHjvHggw+yatUqDhw4wIoVK/jggw+6HOPEiROsXr2a6upqzj77bPrKOn3w\nwQdJSUnh9ttv54477mD79u28+uqrbNy4kccff5wnn3ySl19+mfvuu4+2tjZmzZrF448/TkZGBhdd\ndBEPPfQQZWVl/OIXv+CHP/whU6ZMYfbs2Xi9Xv71X/+1V9vvvvtudu3aRWlpKTfffDN33NH1HY+K\nigqef/557rnnHn70ox/1+TcflaRkw+zLnA84/TxHt3cKz0fPw3tPOtvSC7r28xQuTMh+nnjFfe4G\nNqrqbGBjaDkaLapaGvp0iIwL+CVwo6rOBz4Fbg61PwncjiNChtGFNWvWMGvWLMrLy7n88svZvXs3\n77zzDuXl5Wzbto3Nmzfzxz/+kSlTprB9+3Y++OADrrzySm6//XamTJnCa6+91qvIADQ1NXHRRRex\nbds2MjMzuffee9mwYQO///3v+cd//Meo+3z00Ue89NJLvPPOO3z3u9/F7/dHbffhhx/ywAMP8Oqr\nr7J9+3Z+/OMfh7cdPXqUP//5z6xfv5677+7tX8nhu9/9Lueffz7vvfce1157LQcPHuy17fLly/nT\nn/4EOCG/xsZG/H4/f/7zn7ngggs4fvw43//+93nllVd49913KSsr6/HgP3LkCN/73vfYsmULGzZs\n4KOPPuqyPZrta9as4YILLqC8vJw77riDI0eOcPXVV4f3+cY3vsGDDz6Ia7yErZO8zlhs530dVj8F\nd+6Dv38HVv4YZl3iiNBL34b/uBjWlMCvroNNa2DfJme0kAQgXtJ3HXBRaP6XwCbgrgHumwu0qeon\noeUNwLeBX6hqFVAlItcMn6nGcNOf5zESvPzyy7z88sssXrwYgMbGRnbv3s0FF1zAt771Le666y5W\nrFjBBRdcMOBjJicnc+WVVwKwYMECvF4vHo+HBQsWcODAgaj7XHPNNXi9XrxeLwUFBVRWVlJUVNSj\n3auvvsqqVavIy3PexZg4sbPGyuc+9zlcLhfz5s2jsrKyx76RbN68md/97nfhc0+YMKHXtmeddRbb\ntm2joaEBr9fLkiVL2Lp1K3/605949NFH2bJlCzt37uS8884DwOfzsWzZsi7HeOedd7jwwgvD9n7x\ni1/kk08+CW8fiO1TpkzhhRdeAGD9+vUUFBRw1llnsWnTpj6vdczickH+HOdz1i3OuvojoeSCUILB\n6z90BlEVN0xe2OnxlCyDjIIRNzleQjNJVY8CqOpREentylNEZCvQDqxR1T8AxwGPiJSp6lZgFVA8\nWANE5DbgNoCSkpKhXIMxilFVvv3tb/O3f/u3PbZt27aNF154gW9/+9tcccUVvXoj3fF4POG0WpfL\nhdfrDc+3t0cfWLOjDYDb7e61nar2mrIbeYyBvIA90NRfj8fD9OnTefzxxzn33HNZuHAhr732Gnv3\n7mXu3Lns3buXyy+/nKeeeqrXY/Rnz2Btf+ONN1i7di0vvPACra2t1NfXc9NNN/Hkk08O6JrGLFlT\nYP7nnQ84/TwVf+kUn63/CVtCiTATZ3UVntxZMe/niZnvKSKviMgHUT7XDeIwJaG3Tr8MPCIis9T5\nNt4IPCwi7wANOEI0KFT1MVUtU9Wy/Pz8we5ujEIyMzNpaGgA4LOf/Sz/+Z//SWOjE1o4fPgwVVVV\nHDlyhLS0NG666Sa+9a1v8e677/bYNx5ceumlPPPMM5w4cQKAk92GkR8oy5cv5//9v/8HwIsvvkhN\nTZSXi7u1f+ihh1i+fDkXXHDYEEfYAAAgAElEQVQB//7v/05paSkiwjnnnMMbb7zBnj17AGhubu7i\nrQCcffbZvP7669TU1NDe3s5vf/vbfm3s617/4Ac/oKKiggMHDvD0009zySWXmMhEIyUbTrsMLrkX\nblkPdx+CWzfC5d+D/DPg4xdg7f+Cfz0LXvpOzM2JmUejqpf1tk1EKkVkcsibmQxU9XKMI6HpPhHZ\nBCwG9qrqW8AFoWNdAZw+3PYbY4/c3FzOO+885s+fz1VXXcWXv/zlcKgnIyODJ598kj179nDnnXfi\ncrnweDz89Kc/BeC2227jqquuYvLkyX3208SKM888k3vuuYcLL7wQt9vN4sWLeeKJJwZ9nPvuu4/V\nq1ezZMkSLrzwwn69+QsuuIAHHniAZcuWkZ6eTkpKSjicmJ+fzxNPPMHq1atpa3NqEn3/+9/n9NM7\n/x2nTp3Kd77zHZYuXcqUKVOYN28e2dnZfZ5z4cKFJCUlsWjRIm655Ra+9KUvceutt4bDZ8YQSEp2\n3sspKoPzbnfKaRzf7YTZ8ufE/PRxGetMRP4FOKGqa0TkbmCiqv5DtzYTgGZVbRORPOAt4DpV3Ski\nBapaJSJe4AXgAVV9NWLf+4HGgWadJfzozWOAXbt2MXfu3HibYcSBxsZGMjIyaG9v5/rrr+dv/uZv\nuP7662N+XvvOxZ6BjnUWr7SNNcDlIrIbuDy0jIiUiUhHAv5cYKuIbAdew+mj6XgD7E4R2QW8D6zr\nEBkRKRSRCuCbwL0iUiEiWSN3WYZhdOf++++ntLSU+fPnM2PGDD73uc/F2yRjhLHRmzGPZiQYS78u\nly5dGg4VdfDrX/+aBQsWnPKxT5w4waWXXtpj/caNG8nNjU2Vx3iccyQYS9+5RMVGbzaMGPH222/H\n7Ni5ubmUl5fH7PiJck5jfDGo0JmITBCR2FfJMQzDMMYM/QqNiGwSkazQsDHbgcdFZAyO+2AYhmHE\ngoF4NNmqWg98HnhcVc8Cek1dNgzDMIxIBiI0SaF3XW4A1sfYHsMwDGOMMRCh+SfgJWCPqv5FRGYC\nu2NrlmEMP1aPpiejsR7N9OnTWbBgAaWlpSNScsE4dfoVGlX9b1VdqKpfCy3vU9UvxN40wxheEkFo\nysrKePTRR0/5OP2RCPVo5s2bN6R9B1L47LXXXqO8vBx7LWF00G96s4g8CHwfaAH+CCwCvqGqNsCQ\nMTRevBuO7ei/3WAoXABXremzidWjGRv1aIzRx0BCZ1eEkgFWABU444rdGVOrDCMGWD2asVGPRkS4\n4oorOOuss3jsscf6vF4jMRjIC5ue0PRq4ClVPTnQYcYNIyr9eB4jgdWjGZ31aMApFTBlyhSqqqq4\n/PLLOeOMM1i+fHmf123El4EIzToR+QgndPY1EckHWmNrlmHEFqtH0z+JWI8GHOEBKCgo4Prrr+ed\nd94xoUlwBpIMcDewDChTVT/QhFMh0zBGFVaPZvTXo2lqagpva2pq4uWXX2b+/Pn9HtOILwMZGcAD\n/DXwGxF5FvgqcCLWhhnGcBNZj2bDhg3hejQLFixg1apVNDQ0sGPHDs4++2xKS0t54IEHuPfee4HO\nejQXX3xxXGyPrEezaNEivvnNbw7pOPfddx+bN29myZIlvPzyywOqR3P06FGWLVvGpEmTeq1Hs3Dh\nQs4555wefTCR9Wguu+yyQdejefjhh7v00VRWVnL++eezaNEizj77bK655ppwuNJIXPodvTk0bL8H\n+GVo1V8DAVW9Nca2jRg2enPssZF0xy9Wj2bsMpyjN39GVRdFLL8aqhFjGIbRL/fffz+vvPIKra2t\nXHHFFVaPZhwyEKEJiMgsVd0LEBoZIBBbswwjcbF6NIPjoYcGVOjWGMMMRGjuBF4TkX2AANOAr8TU\nKsNIYKwejWEMjn6FRlU3ishsYA6O0HwElMbaMMMwDGNsMKAKm6raBrzfsSwi/w30na5iGIZhGAyy\nwmYENjSAYRiGMSCGKjQDe4XXMAzDGPf0KjQisk5E1kb5rANik/5iGDEkEcoEWD2aToZaj6a2tpZV\nq1ZxxhlnMHfuXN56661TtteILX310fSVk2j5isaoo0Novva1rw1630ceeYSbbrqJtLS0U7KhrKxs\nRIp1JUI9mqHSITRf/vKXo27/+te/zpVXXsmzzz6Lz+cblh8ARmzpVWhU9fWRNMQYP/zwnR/y0cmP\n+m84CM6YeAZ3nX1Xn22sHs3or0dTX1/P5s2beeKJJwBnxOzk5OQ+/+5G/BlqH41hjDqsHs3or0ez\nb98+8vPz+cpXvsLixYu59dZbaWpq6vOajfgzoPRmwxhO+vM8RgKrRzM669G0t7fz7rvv8pOf/ISl\nS5fy9a9/nTVr1vC9732vz+s24osJjTEusXo0/ZOI9WiKioooKipi6dKlAKxatYo1a+JfSM/om4GU\nCYiWffZrEfm6iKSMhJGGMRxYPZrRX4+msLCQ4uJiPv74Y8AZj23evHn9HtOILwPpo9kHNAL/EfrU\nA5XA6aHlQSMiE0Vkg4jsDk2j+u8iEhCR8tBnbcT6S0TkXRH5QER+KSJJofV/JSLvhz5visiiaMc1\nxidWj2b016MB+MlPfsJf/dVfsXDhQsrLy/nOd74zpHthjBwDqUezWVWXR1snIh+q6pmDPqnIg8BJ\nVV0jIncDE1S1R+BeRBpVNaPbOhfwKXCpqn4iIv8EfKqqvxCRc4FdqlojIlcB96vq0v7ssXo0scdq\ng4xfrB7N2GWg9WgG4tHki0j4Z09oPi+06BuifdfRWUjtl8BgClTkAm2q2uGjbwC+AKCqb6pqRyxg\nC9CzV9UwjBHl/vvvp7S0lPnz5zNjxgyrRzMOGUgywP8H/FlE9uKMcTYD+JqIpNMpFoNlkqoeBVDV\noyJS0Eu7FBHZCrQDa1T1D8BxwCMiZaq6FVgFFEfZ96vAi0O0zzB6xerRDA6rR2MMpEzAC6EyAWcQ\nKhOgqq2hzY/0tp+IvAIURtl0zyDsK1HVI6Fia6+KyA5V3SsiNwIPi4gXeBlHiCLPfTGO0Jzfh323\nAbcB/capDSMSq0djGIOjX6EREQ/wt0BHP80mEfmZqkZ/syyEql7WxzErRWRyyJuZDFT1cowjoek+\nEdkELAb2qupbwAWhY12Bk5jQceyFwM+Bq1T1RB/2PQY8Bk4fTV/XYhiGYQydgfTR/BQ4C/i30Oes\n0LpTYS1wc2j+ZuC57g1EZELIY0FE8oDzgJ2h5YLQ1AvcBfx7aLkE+B3w1xF9OIZhGEYcGUgfzWdU\nNTJN+FUR2X6K510DPCMiXwUOAl8EEJEy4O9U9VZgLvAzEQniCOIaVe0YDvZOEVkRWv9TVX01tP4f\ncZIF/i30Ulr7QDIiDMMwjNgxEKEJiMgsVd0LEOovCZzKSUMhrR69j6HO/VtD828CUXtXVfVO4M4o\n62/t2N8wDMNIDAYSOrsTeE1ENonI68CrOJlohjGqsHo0PRlt9Wg+/vhjSktLw5+srCweeaTXnCQj\nQehXaFR1IzAbuD30mQNYAQhj1JEIQlNWVsajjz56ysfpj0SoRzPUoWH6Epo5c+ZQXl4eHnE7LS1t\nRF7+NE6NAQ2qqaptwPsdyyLy34DlBBtD4tg//zNtu4a3Ho137hkU9jMUidWjGf31aCLZuHEjs2bN\nYtq0aX3+3Y34M9R6NAMb/tUwEgirRzP669FE8vTTT7N69eo+r9dIDIZaJsDeOzGGTH+ex0hg9WhG\nZz2aDnw+H2vXruUHP/hBn9drJAa9Co2IrCO6oAhOCrFhjFqsHk3/JGI9mg5efPFFlixZwqRJkwa8\njxE/+gqdPQT83yifh4CefqxhJDhWj2b016Pp4KmnnrKw2SiiV49GVV8fSUMMI9ZE1qO56qqrwvVo\nwOnIf/LJJ9mzZw933nknLpcLj8fDT3/qDILRUY9m8uTJffbTxIrIejRut5vFixfzxBNPDPo49913\nH6tXr2bJkiVceOGFA6pH88ADD7Bs2TLS09N7rUfTMcjo97//fU4/PTwiVJd6NFOmTBl0PZpbbrmF\nL33pS9x6663h8FlzczMbNmzgZz/72aCv34gP/dajGQ9YPZrYY7VBxi9Wj2bsMpz1aAzDMIaM1aMx\nBpV1JiKFqnosVsYYxmjA6tEMDqtHYww2vfkFYEksDDGM0YLVozGMwTHY0Jm9qGkYhmEMisEKzX/E\nxArDMAxjzDIooVHVoY1IaBiGYYxbLOvMMAzDiCkmNMa4IRHKBFg9mk6GUo8G4OGHH+bMM89k/vz5\nrF69mtbW1lO214gtJjTGuCERhMbq0fRPX0Jz+PBhHn30UbZu3coHH3xAIBDg6aefPhVTjRFgqKM3\nG8aQ+dMzn3D8UOOwHjOvOIMLbji9zzZWj2Zs1KNpb2+npaUFj8dDc3MzU6ZM6fPvbsQf82iMcYPV\noxn99WimTp3Kt771LUpKSpg8eTLZ2dlcccUVfV6zEX/MozFGnP48j5HA6tGMzno0NTU1PPfcc+zf\nv5+cnBy++MUv8uSTT3LTTTf1ed1GfDGhMcYlVo+mfxKxHs0rr7zCjBkzyM/PB+Dzn/88b775pglN\ngmOhM2PcYPVoRn89mpKSErZs2UJzczOqysaNG22E5lGACY0xboisR7Nhw4ZwPZoFCxawatUqGhoa\n2LFjB2effTalpaU88MAD3HvvvUBnPZqLL744LrZH1qNZtGgR3/zmN4d0nPvuu4/NmzezZMkSXn75\n5QHVozl69CjLli1j0qRJvdajWbhwIeecc06PPpjIejSXXXbZoOvRPPzww136aJYuXcqqVatYsmQJ\nCxYsIBgMcttttw3pXhgjh9WjwerRjARWG2T8YvVoxi5Wj8YwjITA6tEYlgxgGIPE6tEMDqtHY5jQ\nGCNGX5lTowmrR5P4WJdAV/y+AE01bTTWttFU20ZjTStNtT4aa1qZsSifuedOjun54yI0IjIR+A0w\nHTgA3KCqPdJfRCQA7AgtHlTVa0PrLwEeApKBbcBXVbVdRK4DvgcEgXbgG6r659hejTEQUlJSOHHi\nBLm5uWNCbIzERVU5ceIEKSkp8TYl5qgqbc3tNNZECkikoDjTtuaeafPetCTSc7z426Kn1A8ncUkG\nEJEHgZOqukZE7gYmqOpdUdo1qmpGt3Uu4FPgUlX9RET+CfhUVX8hIhlAk6qqiCwEnlHVM/qzx5IB\nYo/f76eiosIGQDRGhJSUFIqKivB4PPE2ZcgEg0pznS8kHK1dhCM8rW0j4A923VEgLSuZjBwv6Tle\nZzqhY5oSXu/xuk/ZxoEmA8QrdHYdcFFo/pfAJqCH0PRCLtCmqh0J+xuAbwO/UNXIAbTSAfOfEwSP\nx8OMGTPibYZhJATt/kCvwtGx3FzvQ4NdH2GuJAkLRcG0TGaU5ncKygRnmpadjNudWHle8RKaSap6\nFEBVj4pIQS/tUkRkK04YbI2q/gE4DnhEpExVtwKrgOKOHUTkeuAHQAFwTSwvwjAMIxJVxdfSHlU4\nIqetTT3HtEtOcYcFo3juBDImpHR6JKH1KRmeURl6jpnQiMgrQGGUTfcM4jAlqnpERGYCr4rIDlXd\nKyI3Ag+LiBd4GUeIAFDV3wO/F5HlOP01l/Vi323AbUC/L60ZhmEEg0pLgy8sFl09kc7O9XZfsMe+\nqZkeMiakkJmbwuRZ2V08kI5pcsrYzc2K2ZWpatQHPICIVIrI5JA3Mxmo6uUYR0LTfSKyCVgM7FXV\nt4ALQse6AugxSqOqbhaRWSKSp6o9Kjup6mPAY+D00Qz6Ag3DGDME/MFO76O2laYaX49+kaa6KKEs\nl5Ae8jjyijKYNj+3U0BC69Ozvbg9iRXKGmniJaFrgZuBNaHpc90biMgEoFlV20QkDzgPeDC0rUBV\nq0IezV3AA6H1p+EIkYrIEpystBMjcUGGYSQeqoqvtSO1t7XX/pDWxp6hrCSvOywWU+dM6BHGypiQ\nQmqGB3GNvlDWSBMvoVkDPCMiXwUOAl8EEJEy4O9U9VZgLvAzEQnijGCwRlU7asPeKSIrQut/qqqv\nhtZ/AfgfIuIHWoAvqSXUG8aoRYOK3xeg3RfE3xag3RfA39b5iVzumG/qyNQKCYq/LdDjuCkZnrBg\nFEzP6tGhnjEhheQU96jsD0lEbKwzLL3ZME4FVSXgD+L3BfC3BhxhaAsttwVobwuE5yOX2zsEI4qI\ndMxH6+/oC5dLSMtO7pLWG/ZAcryk56SQnpNMkufUU3uNxE9vNgxjBFFVgu3a+fDv4RkE8be1428L\ndt0WKQjh/bqKSLsvwGB+r4o4YSlPshuP1x2eT05xk57tJcnrwpMcWh/ZLjR1Pq4uyx3z7qTx3ReS\nqJjQGEYCU1vZTHODr4cn4AsvByM8g968B6dNMDi46IUjAK4eD/mUjOTOB36y2xGGbg/87iIS2cad\n5LKQ1DjDhMYwEpDGmjbe+O1u9myNmpAZxu1xRTzwQ8KQ4nRiJ0Wu7+IZOG069+npGSR5XNbJbQwb\nJjSGkUAEAkHe31jBX57fTzColF0znSmzckJi0TNc5DIxMEYBJjSGkSBUfFzD5qc/oeZoE9MX5nH+\nF2eTnZ8ab7MM45QxoTGMONNU28Ybv93D7r9UkpWXwtVfW8iMhXnxNsswhg0TGsOIE4FAkB2vVfDO\n+v0E250w2VmfnUZSsqXeGmMLExrDiANHdtfw+lOfcPJIE9Pm53LBl2aTnZ8Wb7MMIyaY0BjGCNJU\n18abv9vDJ29Xkjkxhav+bgEzFuVZuq8xpjGhMYwRIBgIsmPTYd5Zt4/29iBlV09nyZXT8FiYzBgH\nmNAYRow5sruWzU9/zInDTZTMm8gFXzqdnEkWJjPGDyY0hhEjmut9vPm7PXy85RgZE71c9bcLmFFq\nYTJj/GFCYxjDTDAQ5IPNh3n7uX20+4OcdeU0zrpq+rDUaDeM0YgJjWEMI0f31PL6059woqKR4nkT\nWW5hMsMwoTGM4aC53sdbv9/DR28dI2OClytvm8/MxfkWJjMMTGgM45QIBpUPNx9my3P7aPcFWPLZ\naZRdbWEyw4jEhMYwhsixfXW8/tTHHD/USNEZE1h+4+lMKEyPt1mGkXCY0BjGIGlp8PHW7/ey682j\npOd4+ez/nM+sJRYmM4zeMKExjAESDCo7/+SEyfytARZfUULZ1dNJTrF/I8PoC/sPMYwBcGx/HZuf\n+oTqgw1MneOEySZOtjCZYQwEExrD6IOWRh9bfr+XnW8cJT07mStuPZPTziqwMJlhDAITGsOIQjCo\n7PzzEbb8YS/+1gCll5fwmWssTGYYQ8H+awyjG5UH6tn81MdUfdrA1NNzuODG08mdkhFvswxj1GJC\nYxghWhv9vPXcXnb++QhpWclc/tV5zC6bZGEywzhFTGiMcY8GlZ1vHOGtP+zF1xJg0aXFnH3NDJJT\n7d/DMIYD+08yxjVVn9bz+lOfUHWgnimzc1h+4+nkTrUwmWEMJyY0xriktcnPluf28eGfDpOamcxl\nX5nH6WdbmMwwYoEJjTGu0KCy662jvPW7vbS1tLPo4mI+s3IGXguTGUbMiMt/l4hMBH4DTAcOADeo\nak2UdgFgR2jxoKpeG1p/CfAQkAxsA76qqu0R+30G2AJ8SVWfjd2VGKOJ6oMNvP7Ux1Tur2fyadks\nv3EOeUUWJjOMWBOvn3F3AxtVdY2I3B1avitKuxZVLY1cISIu4JfApar6iYj8E3Az8IvQdjfwQ+Cl\nWF4AQKA9CApujyvWpzJOgdYmP2+v3ccHmw+TmuHhslvmcvrSQguTGcYIES+huQ64KDT/S2AT0YUm\nGrlAm6p+ElreAHybkNAA/xv4LfCZ4TC0L97ZcoR3/+sT3HkpeCenkj41nZzidNIzvaR43KR63KR4\nXKQmd8y78Sa57AE3QmhQ+WjLUd76/V5aG/0svKiIs1fOwJvmibdphjGuiJfQTFLVowCqelRECnpp\nlyIiW4F2YI2q/gE4DnhEpExVtwKrgGIAEZkKXA9cwggIzWGfn794/Ew9EWRSVQvN22uoBo67ghxO\nCnLY7UxrXQohbRGBlKSQAHncpCS7SUlyR4iRK0KknPUpSS5SIsSqY9pxjNTk6OuT3OPX06o+1MDm\npz7m2L56Cmdms/L208kvzoy3WYYxLomZ0IjIK0BhlE33DOIwJap6RERmAq+KyA5V3SsiNwIPi4gX\neBlHiAAeAe5S1UB/XoOI3AbcBlBSUjIIkzq5/qLpXHleMS2+AI1NPqo+rad6fwPZBxspONzIopag\nc64UN658L8FcL205SbRkuGkNBGnxBWjxB2j1B2n1O/M1zT5nnS9Aa3tnm6HgcYsjahEi1ilG7giR\ncnVdFyl4EULYQ9xC6xPJS2tr9vP22v188HoFKRkeLr15LnOWFiKuxLDPMMYjoqojf1KRj4GLQt7M\nZGCTqs7pZ58ngPXdO/dF5ArgVlW9QUT2E/YdyAOagdtCnlCvlJWV6datW4d4NdHRoHLyWBPH9tZx\ndG8dR/fUUn+8FXD6dAqmZTL5tBwmz8qmcGY2Kem9h3NUlbb2TjFq9XcKUFtoXXh9h0h1Wd91n1Z/\noOuxwsIWwB8Y2vchLECRQpYcXdyyUz3kpHnISvWQk+ohJy05vC471UOKZ/DVKVWVj7cc483f7aG1\n0c/8C50wWV/31TCMU0NEtqlqWb/t4iQ0/wKciEgGmKiq/9CtzQSgWVXbRCQPeAu4TlV3ikiBqlaF\nPJoXgAdU9dVu+z9BFGGKRiyEJhpNdW2dwrO3juMHGwgGnfs/YXI6k2dlO8IzK5vs/NS4eAn+QDAs\nTK0RQhXN+wpPu3lfPYTM17muyRegodVPsI+vnTfJFRadnNRkssPznWKU3SFOqR5cdT52PX+Q6v31\nTJqRxYWr55BfYmEyw4g1iS40ucAzQAlwEPiiqp4UkTLg71T1VhE5F/gZEARcwCOq2pFZ9i/AitD6\nn6rqI1HO8QQJJjTd8fsCVB2o5+geR3iO7avD1+JEAVOzkpk8M5vJpznCk1+ciTtpbPS5BINKQ1s7\n9S1+apv91LX4qW3xhefrWvzUNfdcV9vs7xJGTFY4v8XDYp+bVoHNqX4OZrnITosQpAhx6i5a2WnO\nupy0oXlRhjHeSWihSTTiJTTd0aBy8miTIzp76zi6t2u4bdL0LAo7vJ5+wm1jlbb2ALXNPj7ecoyP\nXjqEv6md1LnZBOdnUx8IhESrU5g6RcrXpxeVnOTq6jGldobzOkQpOzLMFxKxrFQPbuv/McYpJjSD\nIFGEJhpNdW0c3dMpPMcPNYbDbROnpIeFZ/KsbLLy4hNuG0lOHG7k9ac+5uieOibNyGL5jadTMC2r\n3/1Ulca29h4eUm2LL+xBdV3XTl2zj9oWP82+vpMxslKSunhHWd3CfDmpyc66tK7rUjyJk0QRL1SV\noEIgqARVCQSVgCrBYOQ8UdY500Cwc3sgqKgqk7JSmJqTist+AMQcE5pBkMhC0x1/Wyjctrc2FG6r\n7xpuCwtPDnklGbjHSIqzr6Wdd9bv5/3XKvCmJrHs+lnMPXfyiGST+dqDnSG9iHBed48p7E2Fw4F+\nAn24UclJri7ekSNCEZ5UmoesFMdr7f6A7Xzo0u2h2/1BTOcDPOJh3rmOqA/4znVEfcB3OX7Uh3/P\n83fZHp7G5m/mTXIxIy+dGXnpzMxPZ2ZeBjPy05mVl0G2vUc1bJjQDILRJDTdiQy3Hd1by7G9deFw\nW5LHRcH0rHCCwWgMt6kqu/9SyRvP7qG5wceZ50/hnOtmkZKR+NehqjT5nFBfbbPf6ZPq1i/V0U/V\npX+qxU9jW3v/J+gHt0twi+ByEZpKxDpn6nZ1296xLrxNcAtd1nXZLoLb1W17xLGcc0Y5fg876OP8\noe0R549+Hc6PjiO1LeyrbmT/8Sb2VTdx8GQz7RGKlpue3ClA+RnMyEtnVn46JRPTSR4j/aAjhQnN\nIBjNQhONptq2LsJTfagRDTovjU7skt2WQ1ZeSsKGb04caWTzU59wZHctBdMyWb56DpOm9x8mGwv4\nA44XVd/iB3p/0IeFJMqD3nDwB4IcOtnMvuom9h13BGhvtSNCxxvbwu1cAsUT05iZl86MvIyQEDne\n0KQsb8L+n8QTE5pBMNaEpjv+tgCVB+o5trfW6e/ZV4ev1el3SAuF2wpnZTP5tBzyiuMfbvO1tvOX\n9fvZ/moFyaluln1uFnPPm2IPT2PYqW/1sz8kQI4QOQK0/3gjrf5guF16spsZ+SEB6haOy/CO35G/\nTWgGwVgXmu4Eg8rJI02O8ITe6Wk4EQq3JUdmt+VQODNrxMYGU1X2bK3ijWd301TvY955UzjnczNJ\nzUgekfMbRgfBoHKsvjXsBXWKUCOHa1uIfGxOyvKGQnFdRahoQuqYHwbKhGYQjDehiUZjTVs41HZ0\nbx3HK7qF20KjGEyelU1m7vCH204eaWLzbz7m8Me15Jdksnz16RTOyB7WcxjGcNDqD/DpiWb2VTeG\nPaAOMaoLhTrBGQKqZGKaI0ChRIQZ+enMzEtnYnrymAjFmdAMAhOanvha20PZbZ0vk/o7wm3ZyeHM\ntsJZ2acUbvO1trP1+QNs33gIT4qbcz43i3nnW5jMGJ2cbPJ1FaBQUsKnJ5rxBTpDcdmpnnBCwqxQ\nQsLM/HSm56aPqpeHTWgGgQlN/zjhtsbwKAZH99bSeNLpSE1KdjFpRlZYeApnZvdbsVJV2bOtijee\n3UNTbRtzz5vMss/NIjXTwmTG2KM9EORwbUsXAXL6gpo4Vt8abicCU7JTewjQzPwMJmelJNwPMBOa\nQWBCMzQaa1o7PZ69dRw/1ODErgVyp6RTOCt6uK3mWBObn/6Eio9qyCvO4MLVcyicaWEyY3zS1Nbu\npGIf7ypA+6obaYp4WTjF42J6bk8BmpGXTnZqfNL9TWgGgQnN8OBrbacyNHbbsb21HNtfHw63pWcn\nUzgrh5T0JHa9eRSP183Sa2dy5vKpCfcrzTASAVWlqqGtSx9QhwAdqmnp8jJwXkaykwXXTYBKJqbF\n9N0gE5pBYEITG4JB5f29LxEAAAhGSURBVMThxogRq51w2xnnOmGytCwLkxnGUPC1Bzl4sjMhITJF\n+0STL9zO7XISEmbkpYcy4jpfUM3PPPV3g0xoBoEJzcjR7g+QNIo6Ow1jtFHX7O/qAUXMt7V3JiRk\neJOYkZfO6rNL+PLSoRV/HKjQjN83jYy4YCJjGLElO83D4pIJLC6Z0GV9MKgcqWvpEoLbd7xpRGwy\noTEMwxgHuFxC0YQ0iiaksfz0/JE994iezTAMwxh3mNAYhmEYMcWExjAMw4gpJjSGYRhGTDGhMQzD\nMGKKCY1hGIYRU0xoDMMwjJhiQmMYhmHEFBuCBhCRauDTIe6eBxwfRnOGi0S1CxLXNrNrcJhdg2Ms\n2jVNVft9+9OE5hQRka0DGetnpElUuyBxbTO7BofZNTjGs10WOjMMwzBiigmNYRiGEVNMaE6dx+Jt\nQC8kql2QuLaZXYPD7Boc49Yu66MxDMMwYop5NIZhGEZMMaEZBCLiFpH3RGR9lG1eEfmNiOwRkbdF\nZHqC2HWLiFSLSHnoc+sI2XRARHaEztmjfKk4PBq6X++LyJIEsesiEamLuF//OEJ25YjIsyLykYjs\nEpFl3bbH6371Z1e87teciHOWi0i9iHyjW5sRv2cDtCte9+wOEflQRD4QkadEJKXb9pg9w6zw2eD4\nOrALyIqy7atAjaqeJiI3Aj8EvpQAdgH8RlX/1wjZEsnFqtpbfv5VwOzQZynw09A03nYB/ElVV4yQ\nLR38GPijqq4SkWQgrdv2eN2v/uyCONwvVf0YKAXnhxZwGPh9t2Yjfs8GaBeM8D0TkanA7cA8VW0R\nkWeAG4EnIprF7BlmHs0AEZEi4Brg5700uQ74ZWj+WeBSEZEEsCtRuQ74lTpsAXJEZHK8jYoHIpIF\nLAd+AaCqPlWt7dZsxO/XAO1KBC4F9qpq95eu4/0d682ueJEEpIpIEs4PhiPdtsfsGWZCM3AeAf4B\nCPayfSpwCEBV24E6IDcB7AL4Qih08KyIFI+ATQAKvCwi20Tktijbw/crREVoXbztAlgmIttF5EUR\nOXMEbJoJVAOPh0KgPxeR9G5t4nG/BmIXjPz96s6NwFNR1sfrO9ZBb3bBCN8zVT0MPAQcBI4Cdar6\ncrdmMXuGmdAMABFZAVSp6ra+mkVZF9OUvgHatQ6YrqoLgVfo/MUSa85T1SU44Yu/F5Hl3baP+P0K\n0Z9d7+IMq7EI+AnwhxGwKQlYAvxUVRcDTcDd3drE434NxK543K8woXDetcB/R9scZd2IpNn2Y9eI\n3zMRmYDjscwApgDpInJT92ZRdh2W+2VCMzDOA64VkQPA08AlIvJktzYVQDFAyDXNBk7G2y5VPaGq\nbaHF/wDOirFNHec9EppW4cSoz+7WJHy/QhTR05UfcbtUtV5VG0PzLwAeEcmLsVkVQIWqvh1afhbn\nAd+9zUjfr37titP9iuQq4F1VrYyyLS7fsRC92hWne3YZsF9Vq1XVD/wOOLdbm5g9w0xoBoCqfltV\ni1R1Oo47/Kqqdv81sBa4OTS/KtQmpr+eBmJXt5j0tThJAzFFRNJFJLNjHrgC+KBbs7XA/whlBp2D\n48ofjbddIlLYEZcWkbNx/kdOxNIuVT0GHBKROaFVl8L/3979hVZdh3Ecf3/SLpzKqJGCF7kLBSOK\nESGBsYtWF3kRKLtQ58VGFyvEguimkCQFkyDqKl1EOcpAEAIhEWVo4QaVmjgakaNglxJ40doItj1d\nfL8HDnOL03Hfc2Z+XjD4nd/fZ184v+ec7znneRibt1vDx6uWuJoxXvPsZvHpqYaPWS1xNWnMJoBn\nJLXka3dx572g2D3M3zq7C5IOAVci4gzpA9MvJI2TXgXsWiZxvSbpJWAmx9XbgBDWA1/n59JK4KuI\nOCfpFYCIOA6cBbYD48AU0LdM4uoGXpU0A0wDu0q/YMj2AyfzlMtvQN8yGK9a4mrWeCGpBXgB6K9a\n1/QxqyGuho9ZRHwv6TRp2m4G+An4pFH3MFcGMDOzojx1ZmZmRTnRmJlZUU40ZmZWlBONmZkV5URj\nZmZFOdGYmVlRTjRm9xClNgd1/YpcqWXEhqU4l9l/4URjdv/oJdW5MmsoJxqzOkhqV2oG9qlSI6mT\nkp6XNCzppqSt+W8kVz4eqZRykfSGpM/y8hP5+IX6vCCpTdL5fI4BqgofStor6Qel5lkDSv1PkDQp\n6QNJ1yQNSXpEUjfwNOlX/tclrcqn2Z/3G5W0peSY2f3LicasfptIjcGeBLYAe4BngTeBt4FfgM5c\n+fgd4Eg+7iNgk6QdwOdAf0RMLXKNg8DlfI4zwKMAkh4jNaXaFhEdwCzQk49ZTSro+BTwLXAwIk4D\nV4CeiOiIiOm87x95v2M5brMl51pnZvX7PSJGAST9DAxFREgaBdpJ1W8HJW0mlVt/ECAi5iT1AjeA\ngYgY/pdrdAI783HfSLqd13eRKnH/mGu3rQJu5W1zwKm8/CWpUu9iKtuuVq5jttScaMzq93fV8lzV\n4znSc+swcDEidij1X79Utf9mYJLaPjNZqCChgMGIeKvO4ysqMc/i+4EV4qkzs3JaST3joapqtqRW\n0pRbJ9CWPz9ZzHfkKTFJLwIP5fVDQLekdXnbw5I25m0PkCoEQ5rOu5yX/wTW3sX/Y1YXJxqzct4H\n3pM0DKyoWv8h8HFE/Aq8DBytJIwFvAt0SrpG6p8zARARY8ABUlvqG8AFoNJ76C/gcUlXgeeAQ3n9\nCeD4vC8DmBXnNgFm/zOSJiNiTbPjMKvwOxozMyvK72jMlgFJfcDr81YPR8S+ZsRjtpScaMzMrChP\nnZmZWVFONGZmVpQTjZmZFeVEY2ZmRTnRmJlZUf8AonhHWSw+F/wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x264bc0ed550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch.best_score_, gsearch.best_params_))\n",
    "test_means = gsearch.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch.cv_results_).to_csv('my_preds_maxdepth_min_child_weights_2.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(min_child_weight), len(max_depth))\n",
    "train_scores = np.array(train_means).reshape(len(min_child_weight), len(max_depth))\n",
    "\n",
    "for i, value in enumerate(min_child_weight):\n",
    "    pyplot.plot(max_depth, test_scores[i], label= 'test_min_child_weight:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'max_depth' )                                                                                                      \n",
    "pyplot.ylabel( '- Log Loss' )\n",
    "pyplot.savefig( 'max_depth_vs_min_child_weght2.png' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
